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EXECUTIVE SUMMARY 



In the Fall of 2010, the Bureau of Indian Education (BIE) approached Northwest Evaluation Association (NWEA) to 
commission a report of system-wide performance for their first full school year of testing as part of the BIE-NWEA 
Project. Even with only a single year of testing data, we were able to identify some patterns and they are listed in this 
report. More years of data will deepen the analyses that are possible and allow for the identification of trends and 
change over time. 

This report is not an evaluation of BIE effectiveness; it is a written summary of data from the BIE's first year of the NWEA 
Project. As a baseline report, it should be used as a place to gauge the starting point for the many schools managed by 
the Bureau, not as a reward or punishment system for schools. This report is a description of first year performance, not 
of BIE effectiveness. One of our recommendations included later in the report is that the BIE not only continue annual 
reporting but also commission a system-wide evaluation of effectiveness after at least three years of testing with NWEA. 

The Findings section of this report describes the Fall 2009 starting point for students in different groups, as well as the 
Spring 2010 end point for this analysis. Growth was measured compared to national NWEA norms and also compared to 
a group of students with similar scores, similar school poverty, and similar rural location (Virtual Comparison Group or 
VCG). Differences were shown for Project schools, non-Project schools, different grades, and to some extent differences 
for ADDs, ELOs, and states. The following is a summary of the report findings. 

• Fall starting scores for BIE students lagged significantly behind national norms in Math and even more so in 
Reading. In general, Fall to Spring growth averages were lower for BIE students than the national norm sample 
of students with similar scores. 

• BIE students averaged across all grades had similar performance to their Virtual Comparison Groups (VCG). BIE 
students were generally lower than their VCG group in early grades K-7 and higher than their VCG group in later 
grades 8-12. 

• For students in the BIE-NWEA Project, students started at lower scores and showed mixed results for growth. 
Project and non-Project students showed similar growth scores in Math but non-Project students showed more 
growth in Reading. There was variance by grade. 

• Average growth for students in Oregon and Utah was greater than comparison group growth in Reading and 
Math. Students in Standing Rock ELO and New Mexico Navajo North ELO had higher student growth scores in 
Reading and Math than their comparison groups. 

• Students in schools that used NWEA professional development showed more growth than comparison students 
in some grades. Students in schools that have been testing with NWEA for 2 years of more showed higher 
growth than comparison groups in early grades. Students in schools managed by the Bureau showed 
significantly greater growth than students in grant or contract schools when compared to both the norms and 
their respective Virtual Comparison Groups. 

The data from this report reflects the academic performance for BIE students and the necessary challenge for BIE 
schools to overcome. Within the data, however, there were several areas with remarkable outcomes that can be 
highlighted and should be celebrated and continued. Looking at high-level averages often masks the success stories for 
some students and schools. Some examples of individual successes are listed in the Discussion section. 
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After analyzing the performance data for the 2009-2010 school year, we have the following recommendations for the 
Bureau of Indian Education. These recommendations are detailed in the report. 

• We recommend that the Bureau continue to look closely at its student performance data on a regular basis. 

• We recommend that the Bureau undertake further analysis in areas not pursued in this report. 

• We recommend that the Bureau continue to monitor progress on an annual basis for several years, and then 
commission a program evaluation study including at least three years of student performance. 

• We recommend that the Bureau use their data to ask difficult questions about student performance and school 
programs. 

• We recommend that the Bureau celebrate and build on successes. 

We hope that the data from this report helps the Bureau of Indian Education understand the current state of their 
student achievement. However, data is the merely the beginning of process improvement. Answering the questions 
raised by the data and implementing necessary changes is what brings about a true increase in student achievement. 
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INTRODUCTION 



THE PURPOSE OF THIS REPORT 

In the Fall of 2009, the Bureau of Indian Education started partnering with Northwest Evaluation Association (NWEA) to 
provide professional development and leadership coaching in 46 schools in seven states. The partnership focused on 
data derived from the use of NWEA's interim assessment, Measures of Academic Progress (MAP) to measure student 
progress and improve school outcomes. Due to the number of schools across states and Line Offices using the MAP 
assessment, the BIE asked NWEA to develop an approach to look at the system of participating BIE schools at once. 
"Roll-up reporting" was provided for the entire system in a way that BIE, ADD, and ELO leaders, as well as teachers and 
school leaders, could look at results and make appropriate choices about curriculum and instruction. Those "roll-up 
reports" were provided after the Spring 2010 testing season and again after the Fall 2010 term when an expanded group 
of 53 schools were participating in the partnership. While 53 schools are included in the professional development and 
leadership coaching project, 112 BIE schools across the country use NWEA MAP assessments. In Fall 2010, BIE 
approached NWEA to commission a report of system-wide performance for their first full school year. This report serves 
as the first in what we recommend to be a series of ongoing reports on the progress of BIE schools. 

With only a single year of testing data, certain patterns were able to be identified and are detailed in this report. 
However, more years of data will deepen the analyses and allow for the identification of trends and change over time. 
Intervention strategies often take several years for results to be seen at a high level (although teacher-student changes 
can often be seen very quickly), so a multi-year evaluation is the most appropriate methodology to reflect the result of 
Bureau effectiveness. However, this report is not an evaluation of BIE effectiveness. Instead, it is a written summary that 
shows data for the BIE's first year of testing with NWEA. As a baseline report, it should be used as a place to gauge the 
starting point for the many schools managed by the Bureau. 

It is important to note this report is a high-level summary of all schools within the BIE. Due to confidentiality, we have 
specifically not identified individual schools, teachers, or students, and the results shown reflect only average 
performance for the different groups identified. An average always masks the individual differences seen for particular 
schools, teachers, and students. Even if average performance falls in a certain range, there are students who are 
struggling as well as students achieving far higher than the average conveys. Education is about student learning, not 
system-wide averages, therefore this report is not designed to be used as a reward or punishment system for schools or 
administrators. There are other tools available to measure school and teacher effectiveness. This report is a description 
of first year performance, not of BIE effectiveness. One of our recommendations included later in the report is that the 
BIE continue annual reporting, but also commission a system-wide evaluation of effectiveness following the third year of 
testing with NWEA. 

THE BUREAU OF INDIAN EDUCATION (BIE) 

Through the design and execution of effective education programs, BIE contributes to the development of quality 
American Indian and Alaska Native communities. Approximately 4,300 full-time and seasonal BIE employees, including 
teachers, serve American Indian and Alaska Native students at BIE-operated schools located on or near Indian 
reservations. 

The BIE supports education programs and manages residential facilities for Indian students at 184 BIE-funded 
elementary and secondary schools and dormitories, 60 of which are BIE-operated schools, and 124 of which are BIE- 
funded tribally-operated schools. BIE’s elementary and secondary school system spans 23 states serving diverse Indian 
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communities. Schools range in size from eight to more than 1,100 students, representing over 240 Tribes with different 
cultural backgrounds and on 64 reservations. 

The BIE's school system is designed to meet the Federal government's commitment to provide for the education of the 
American Indian/Alaskan Native children as called for in numerous treaties, court decisions, and legislation. Achieving 
AYP is one of the cornerstones of the Federal NCLB. In FY 2005, the BIE implemented several provisions required in NCLB 
that were developed through a successfully negotiated rulemaking process in 2004 with Indian tribal leaders. One of 
these provisions addresses AYP standards for student achievement. Consensus was reached that BIE schools would use 
the same AYP standards as the state within which they are located. Application of this methodology allows BIE to track 
student academic proficiency in each of the BIE-funded elementary and secondary schools relative to local public school 
performance. 

THE NWEA-BIE PROJECT 

NWEA is a not-for-profit organization located in Portland, Oregon with offerings in computer-adaptive assessment, 
research, professional development, and reporting. NWEA's Measures of Academic Progress (MAP™) assessments are 
aligned to state standards and can predict proficiency on state exams. 112 BIE schools in 17 states use MAP assessments 
to inform instruction and predict proficiency on state tests. 

Through a professional development contract, NWEA works closely with School Leadership Teams and Education Line 
Officers in 53 schools and 17 line offices within the BIE's System of Support. Between program initiation in September 
2009 and January 2011, NWEA has delivered more than 175workshops, 200 face-to-face leadership coaching events, 
and 220 telephone coaching events to principals and Education Line Officers. "Roll-up" Reports with aggregated student 
achievement and growth data have been delivered to BIE at the conclusion of each testing season. 

The professional development and coaching plan is designed to work seamlessly with the BIE System of Support to 
provide structures and build capacity for the application of MAP student achievement data to improve effectiveness in 
BIE schools and make a difference for children. The plan is designed to support and enhance BIE's "commitment to 
provide high-quality professional development opportunities" and help "ensure delivery of scientifically researched and 
evidence-based instructional practices." 

The long-term partnership between NWEA and the Bureau provides an environment for the development of capacity 
building and proficiency in the use of student achievement data over an extended time. The workshops delivered at 
schools are "leveled" over time to indicate a progression in complexity in the acquisition and development of skills and 
core knowledge. Through this process, BIE personnel develop capacity to access and apply data to inform planning, 
implementation, and evaluation of student outcomes. 

To accomplish this work, NWEA works closely with teachers, school leadership teams, Education Line Officers, and the 
BIE System of Support Management Team. During the past year and a half, many collaborative relationships between 
NWEA coaches and leaders have been established. The consistent support provided to BIE leaders has resulted in 
changes of practice in schools and line offices, with leaders now eager to set and monitor goals. This same staff has 
articulated, in many venues, the value of these robust coaching relationships. 

MEASUREMENT ISSUES 
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Federal Adequate Yearly Progress (AYP) measures the percentage of students in each state meeting the state-defined 
grade-level proficiency bar. There are a number of reasons why this metric is may not be useful for measuring BIE 
students. 

For instance, measuring student success with AYP ratings: 

• Can't compare state to state— it is meant for measuring students in one state only 

• Can't measure growth over time in states without a vertical scale (e.g., California)— it is meant for grade-specific 
measurement only in those states 

• Can't measure students above or below grade level in states disallowing off-grade testing —it is meant for 
typical students only 

One of the challenges facing the Bureau of Indian Education is a lack of comparable data for its multiple schools in 
multiple states. For instance, since each state has a different proficiency bar, a student who received the same score in 
two states could be considered passing in one and failing in another. This means that comparing AYP measures across 
states is likely to distort conclusions drawn from examining two or more states at the same time. 

Another problem with the easily available AYP rating is it simply reports students as Pass or Fail, not measuring progress 
they have made because of a lack of reliability at the student level. Even in states starting to use growth as a measure of 
success, the tests they are using were primarily designed to measure growth at the aggregated levels (classroom, school, 
district, and state), not individualized growth. In addition, for students significantly above or below grade-level 
proficiency levels, state tests often do not measure performance as accurately as a computer adaptive instrument. 

Those tests often don't have the number of questions needed to accurately measure a student outside typical on-grade 
performance. 

NWEA tests address many of these concerns because the MAP test measures individual student growth on a single scale 
from Kindergarten through high school in a manner comparable across states. With reporting tools designed to harvest 
these data, the Bureau is able to see student growth between different schools, as well as across different grades within 
their schools. This report uses test scores from students in BIE schools and compares their students' performance 
against the performance of students across the nation, providing a description of the progress of the Bureau's 
educational system. What follows is an explanation of how this progress was assessed, what progress was found, and 
how the progress compares to other types of students. 
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METHODOLOGY 

SCHOOLS AND STUDENTS 

Data available for this project included Fall 2009, Spring 2010, and Fall 2010 NWEA Reading and Math scores for 
students in some of the BIE schools. Some BIE schools started using the NWEA tests in 2000, and other schools have 
been joining in every season and year in between. In the fall of 2009, 73 of 184 BIE schools tested with NWEA, and in the 
fall of 2010 that number increased to 112. Testing was not uniform in all schools. Not all schools tested all grades, and 
some schools did not test during every term (fall, winter, spring). While some schools used NWEA professional 
development services to understand their data and how to use it, others haven't used these services. Because there is 
such variation between which schools and students were included in different pieces of the dataset, we developed a 
core group of student test records that would be used for this report. The core group includes students who had valid 
test records in either Reading or Math or both in Fall 2009 and Spring 2010, and were in the same school in both 
seasons. In addition, because disaggregation was necessary between the NWEA Project schools and the non-Project 
schools, analysis for those populations was limited to students in grades K-8. Grades 9-12 had less than 100 NWEA 
Project students per grade so disaggregation by this group wouldn't be possible. 

Tables 1 and 2 show the number of all BIE students with valid scores in each term and subject. The column to the right of 
the dark blue line in each table shows the total number of students included in this report's analysis. The last column 
shows what percentage of Fall 2009 students were included in this report. Most students who were not included 
weren't included because they did not have a Spring test score, but some were not included because they were not in 
the same school in Spring. 



Table 1: Counts of BIE students in each testing season for Math 



MATH 


Fal 109 Data 


Spri nglO Data 


Fall 10 Data 




Total 

Included 


% of Fa II 09 
Included 


K 


635 


1176 


1219 


488 


77% 


1 


852 


1361 


1624 


598 


70% 


2 


1191 


1658 


1863 


995 


84% 


3 


1175 


1808 


2041 


994 


85% 


4 


1218 


1819 


1972 


1008 


83% 


5 


1144 


1718 


1990 


937 


82% 


6 


1153 


1732 


1912 


934 


81% 


7 


1176 


1467 


1750 


881 


75% 


8 


1151 


1393 


1680 


869 


75% 


9 


1069 


1138 


1646 


650 


61% 


10 


856 


1059 


1306 


541 


63% 


11 


686 


873 


1119 


458 


67% 


12 


330 


442 


708 


198 


60% 


Total 


12,636 


17,644 


20,830 




9,551 


76% 
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Table 2: Counts of BIE students in each testing season for Reading 



READING 


Fal 109 Data 


Spri nglO Data 


Fa 1110 Data 




Total 

Included 


% of Fa II 09 
Included 


K 


631 


1176 


1219 


511 


81% 


1 


831 


1361 


1624 


625 


75% 


2 


1228 


1658 


1863 


1000 


81% 


3 


1201 


1808 


2041 


990 


82% 


4 


1205 


1819 


1972 


992 


82% 


5 


1135 


1718 


1990 


911 


80% 


6 


1124 


1732 


1912 


923 


82% 


7 


1176 


1467 


1750 


907 


77% 


8 


1124 


1393 


1680 


853 


76% 


9 


1075 


1138 


1646 


675 


63% 


10 


863 


1059 


1306 


594 


69% 


11 


711 


873 


1119 


485 


68% 


12 


340 


442 


708 


193 


57% 


Total 


12,644 


17,644 


20,830 




9,659 


76% 



NWEA TESTING 

NWEA tests are administered at schools during specific testing windows in fall, winter, and spring. BIE schools can 
choose when to administer the test, but tests must be administered within a specific window in order to accurately 
measure growth between testing events. The assessments use computer adaptive testing (CAT) technology which means 
questions are selected for the student based on their performance on previous questions. This allows for a more precise 
measurement of student performance than traditional paper-and-pencil tests. Most subject tests consist of about 50 
multiple choice questions. Either before the student takes the test or afterward, a school representative enters the Class 
Roster File for each group of students. This includes the student's identifying information as well as the school they are 
in. The Class Roster File also allows schools to enter teacher names and special population information (such as whether 
a student is an English Language Learner, or whether the student is in a specialized math program) so that schools can 
disaggregate their data by these categories later. The testing information as well as the identifying information is 
warehoused at NWEA so data can be reported back to schools and used for analysis and coaching services requested by 
the school. 

NWEA test scores are called RIT scores because they are developed using Rasch methodology for measuring student 
growth. NWEA has been using this method of measuring students for over 30 years and has developed cross-grade 
scales for each subject tested that have remained stable during the course of this time. These scales are used for all 
NWEA students using the Measured Assessment of Progress (MAP) and MAP for Primary Grades (MPG). NWEA tests are 
aligned to the content standards of each state, but are drawn from a single pool of calibrated items. Because each 
state's NWEA test is a part of the same national assessment, student performance in each subject can be compared 
across states. 

This makes NWEA tests particularly good for measuring BIE students who are located in many states and whose 
performance is measured based on the different standards in each of their states. Using NWEA RIT scores, the Bureau of 
Indian Education can compare performance and growth for all of their students across the country. NWEA tests are 
designed for the specific purpose of measuring student academic growth, and unlike most state assessments, they are 
equally valid for all students being tested. The computer-based tests adapt to each student's ability level, so students 
who are far above or far below grade level can be measured just as effectively as average students. A single point of RIT 
growth means the same amount of learning no matter what grade the student is in. 
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MEASURING PERFORMANCE 



Testing data provided by NWEA allows for multiple ways to measure student performance. We have used four main 
metrics in this report: test scores, raw growth, norm-indexed growth, and virtual comparison group measurement. 

Test Scores 

The most basic measurement is to evaluate the student's status at a certain time by looking at their test score. NWEA 
tests have an advantage over most other tests used in districts and states because all test scores are measured on a 
single scale across all grades and all states. This scale is called a RIT (Rasch UnIT) scale, and the scores provided are called 
RIT scores. RIT scores range from about 100 for early learners (often Kindergarten and 1 st grade) to as high as 300 for the 
most advanced learners (often 11 th and 12 th grade), with the majority of scores being between 150 and 250. The scale is 
cross-grade, meaning that student scores are reported on the same scale over time, rather than a separate scale at each 
grade. Most analyses in this report have been based on grade level because that is the most convenient way for schools 
and administrations to view performance. RIT scores from Fall 2009 and from Spring 2010 are reported in this study. 

Raw Growth 

One of the differences of NWEA tests and state accountability tests is that NWEA tests were designed to measure 
student growth. The difference in RIT scores between two periods (for example between Fall and Spring of a school year, 
or between one year and the next) is called Raw Growth and reflects the amount of domain (for instance Math or 
Reading) learning the student has gained between the two periods. Raw growth can be compared for groups of students 
to see which group gained more growth over the same period of time. 

Growth Norm Index 

One way to give context to a raw student growth measure is to compare the raw growth of one student to a national 
sample. For this report, we have compared the growth of BIE core students to a national sample of NWEA students in 
the same grade and subject who had the same starting RIT score. This is called a Growth Norm comparison. The NWEA 
Growth Norms are comprised of test events for millions of students in thousands of districts across the country. They are 
balanced to reflect the national demographic distribution of students, and they are revised every few years. The 
advantage of using a Growth Norm comparison instead of looking only at raw growth is that students at higher RIT score 
levels typically show lower growth than other students. Limiting the comparison to students with the same starting 
score produces a more apples-to-apples measurement. Subtracting the Growth Norm from a student's raw score creates 
what we call a Growth Index Score based on the norm. Values above zero show that a student has grown more than a 
typical group at that starting RIT score, and values below zero indicate that they have grown less. Growth Index scores 
can be compared across all grades which make it possible to see which grade levels need more interventions than 
others. Because state assessments are not always "vertically scaled" like NWEA tests, comparing across grades in those 
tests is not feasible. For more information about the problem of using state assessments to measure performance and 
growth across grades, please see The Proficiency Illusion (Cronin, Dahlin, Adkins, & Kingsbury 2007) . 
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Virtual Comparison Group 



Another way of understanding student growth is by comparing an individual student's raw growth to the average raw 
growth of a group of similar students: we call this method a Virtual Comparison Group (VCG) analysis. For every student, 
we create a sample of 51 students across the country who have the same starting RIT score, and in addition have the 
same school characteristics as the student being measured: all students in the VCG group come from schools with the 
same percentage of Free and Reduced Lunch students (as a measure of poverty rate) and the same geographic 
categorization (Urban, Suburban, Town, or Rural) as the student being compared. In this way, VCG analysis uses an even 
more precise comparison group to compare student growth. The advantage of this method is that it further equalizes 
the measurement for students who come from poor and rural schools. VCG averages include students from across the 
country but deliberately exclude all BIE students. More information about how norm indexed growth and VCG growth 
analyses are constructed is included in the Appendix. 

LIMITATIONS 

As with any research study, there are limits to the data that was available, which in turn limits how the study can be 
used. In order to measure student growth scores by school, students who were not in the same school in both seasons 
were not included. This makes comparisons more valid, but it means that the performance and growth of students who 
changed schools in the middle of the year are not included. Because mobile students often have lower scores, this 
introduces a slight skew in the dataset. In addition, only schools that were testing with NWEA in both Fall 2009 and 
Spring 2010 are included in this report. Many schools started testing with NWEA in Spring 2010 or Fall 2010, and could 
not be included in the growth analysis between Fall 2009 and Spring 2010. One other limitation of the dataset is that all 
students were measured based on their grade level in the Fall of 2009. Students who skipped to the next grade or who 
failed a grade are included with other students who started in the same grade, meaning that the school interventions of 
changing a student's grade based on need is not captured in this data. 

In addition, there may be differences in how different schools administer the NWEA tests. Some schools use these tests 
as an integral part of their assessment program; others may not yet have fully implemented NWEA testing. This may 
affect the reliability and validity of results. In addition, administration of the Project may vary across BIE settings. Some 
schools have had more time working with NWEA, and some schools may have had more professional development 
courses or spent more time with school coaches. These differences in rigor of Project administration may also cause 
variance in the results shown in this report. 

The most important limitation of this study is the lack of historical perspective. Because this is the first year of the BIE- 
NWEA Project, we could only use data from a single school year. Growth within a school year is very useful for classroom 
and school administration practices, but isn't indicative of a system-wide trend. Fall to Spring growth is a single point of 
growth data, and more stable results can be expected as more growth data is collected. This report should not be used 
to measure whether the BIE school system is effective or not. This report offers data about the baseline performance 
and growth achieved by schools, and is not a summative evaluation of the Project. We recommend a follow-up report in 
two years to measure the effectiveness of the Bureau's schools, as well as the effectiveness of the BIE NWEA Project. 
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FINDINGS 



OVERALL DIFFERENCES 

Because reliable growth measures were needed, the majority of the analysis in this report is concentrated on a core 
group of students who had the following characteristics: 

• They were in the same school in Fall 2009 and Spring 2010. 

• They had valid scores in Fall 2009 and Spring 2010 in either Reading or Math or both. 

In addition, division by NWEA Project schools versus non-Project schools is limited to grades K-8 because there weren't 
enough students in the upper grades in Project schools to make the comparison reliable. NWEA norm comparisons are 
limited to grades 2-11. 

For more information about the selection of core students, please see the Methodology section. The following tables 
show the counts of BIE students in NWEA Project and non-Project schools for each subject and grade, as well as the 
average RIT score from Fall 2009 which serves as the baseline from which we measured growth. The average Fall RIT for 
students' Virtual Comparison Groups (VCG) is shown next to the Project schools' and non-Project schools' average RIT 
scores; note that the comparison groups for Project and non-Project schools are different. The national NWEA 
normative score (Norm) for Fall is also shown. BIE average and VCG averages for Fall RIT should be very close since the 
comparison groups were chosen in part based on this score. 

Table 3: Fall Math RIT scores for Project, non-Project, VCG, and Norm groups 



MATH 


Number of Students 


Project 


Non- 

project 


K 


226 


262 


1 


329 


269 


2 


383 


612 


3 


343 


651 


4 


379 


629 


5 


333 


604 


6 


360 


574 


7 


299 


582 


8 


273 


596 



10 



11 



12 



61 



67 



63 



44 



Total 3160 



589 



474 



395 



154 



Fall RIT Score 



Project 



131.5 



153.7 



170.6 



180.9 



189.8 



197.7 



204.1 



208.8 



214.1 



VCG 



133.2 



153.7 



170.5 



181.6 



190.1 



197.8 



204.7 



208.7 



213.9 



Fall RIT Score 


Non- 

project 


VCG 


132.9 


134.81 


152.8 


154.61 


172.4 


172.4 


183.4 


183.4 


193.8 


193.8 


201.6 


201.8 


208.0 


208.1 


213.2 


213.7 


218.4 


218.4 




179.5 



182.1 



203.0 



211.7 



218.3 



224.1 



229.3 
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Table 4: Fall Reading RIT scores for Project, non-Project, VCG, and Norm groups 



READING 


Number of Students 


Project 


Non- 

project 


K 


243 


268 


1 


340 


285 


2 


384 


616 


3 


349 


641 


4 


376 


616 


5 


336 


575 


6 


357 


566 


7 


302 


605 


8 


267 


586 



10 



11 



12 



58 



68 



64 



44 



Total 3188 



617 



526 



421 



149 



Fall RIT Score 



Project 



132.8 



151.7 



164.9 



174.8 



183.7 



189.5 



195.8 



199.4 



202.2 



VCG 



134.9 



151.7 



165.4 



176.0 



183.8 



189.6 



196.2 



199.6 



202.7 



Fall RIT Score 


Non- 

project 


VCG 


134.0 


134.61 


150.0 


152.91 


168.3 


168.3 


179.6 


179.6 


189.1 


189.0 


195.8 


195.8 


200.3 


200.3 


205.2 


205.3 


209.0 


209.1 




179.7 



191.6 



200.1 



206.7 



211.6 



215.4 



219.0 




As is evident in the tables above, the average starting RIT score for Project students was slightly below the average 
starting RIT score for non-Project students in all subjects and all grades except 1 st . This is likely due to the fact that BIE 
chose schools for the NWEA Project based on low AYP performance. The average starting RIT score for all BIE students 
(Project and non-Project) and their VCG group was almost the same average score since each comparison group was 
chosen based on the student's starting RIT score, as well as their school poverty rate and their geography. For more 
information about the Virtual Comparison Group (VCG) comparison, see the Methodology. Both BIE students and their 
VCGs were significantly below the national average. 

The performance of these different groups of students has been measured with several metrics. Fall 2009 RIT is the 
baseline for this report. Spring 2010 RIT is the endpoint for this analysis. Raw growth is the difference between their Fall 
RIT and their Spring RIT. Norm-indexed growth measures their growth against a national standard. VCG analysis 
measures their growth against a group of similar students. 

The following charts show the average performance on Spring 2010 RIT scores for Reading and Math for NWEA Project 
students, non-Project students and the NWEA norm students. A table of values for each group follows the two charts. 
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Figure 1: Spring Math RIT Scores 
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Figure 2: Spring Reading RIT Scores 



Spri 

BIE Pr 


ng2010 Reading RIT Scores 

oject and NonProject Schools Compared to NWEA Norms 

■ Project Nonproject — NWEA Norm 




_ 


<U 

o 




to^UU - 

1- 

cc 

o 

t-H 

o 

fN 

CUD 

.E 175 

Q. 

l/D 

150 -- 
125 


K12345678 



14 | P a g e 



Table 5: Spring RIT scores for Project, non-Project, all BIE, and Norm groups 









Math 










Reading 








Grade 


Project 


Non-Project 


All BIE 


Norm 




Grade 


Project 


Non-Project 


All BIE 


Norm 




K 


144.1 


148.0 


146.2 






K 


138.3 


148.3 


143.6 






1 


163.6 


168.5 


165.8 






1 


163.3 


163.7 


163.5 




Spring 


2 


181.3 


184.3 


183.1 


190.8 




2 


176.2 


180.1 


178.6 


189.6 


2010 


3 


190.6 


194.4 


193.1 


202.4 




3 


181.7 


188.1 


185.9 


199.0 


RIT 


4 


198.2 


202.9 


201.1 


211.6 




4 


190.2 


196.0 


193.8 


205.8 




5 


205.3 


208.5 


207.4 


219.2 




5 


194.2 


201.6 


198.9 


211.1 




6 


210.9 


214.2 


212.9 


223.8 




6 


199.6 


205.0 


203.0 


214.8 




7 


213.4 


218.3 


216.7 


228.3 




7 


203.3 


209.1 


207.2 


217.9 




8 


219.6 


224.0 


222.6 


232.7 




8 


206.7 


212.1 


210.4 


221.2 



The charts and table above show that Spring scores of non-Project students are still above Project student scores. BIE 
scores for both groups are 7-14 points behind national norms. Differences varied substantially by grade and subject: 
Project students lagged behind non-Project students by 10 points in Kindergarten Reading but only 3 points in 5 th grade 
Math. 

The next set of charts shows the differences in raw growth between the different groups of students. 

Figure 3: Math Fall to Spring Raw Growth 
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Figure 4: Reading Fall to Spring Raw Growth 



Fall to Spring Raw Growth Reading 

BIE Project and NonProject Schools Compared to IMWEA Norms 

■ Project Nonproject —Norm 

20 -i 




K12345678 



Table 6: Fall to Spring Raw Growth for Project, non-Project, all BIE, and Norm groups 





Math 


Reading 




Fall- 


Grade 


Project 


Nonproject 


All BIE 


Norm 




Grade 


Project 


Nonproject 


All BIE 


Norm 


K 


15.7 


15.0 


15.3 




K 


13.3 


14.5 


13.9 




1 


13.6 


15.9 


14.5 




1 


11.8 


12.1 


12.0 




2 


10.8 


12.1 


11.6 


13.2 


2 


11.1 


11.7 


11.4 


12.5 


Spring 


3 


9.2 


11.2 


10.5 


10.7 


3 


6.6 


8.6 


7.9 


8.8 


Raw 

Growth 


4 


8.3 


9.2 


8.9 


8.9 


4 


6.5 


6.9 


6.8 


6.6 


5 


7.6 


6.8 


7.1 


7.3 


5 


4.8 


5.8 


5.4 


5.1 




6 


6.3 


6.2 


6.2 


6.4 


6 


3.7 


4.8 


4.4 


4.3 


7 


5.0 


4.9 


4.9 


6.0 


7 


3.9 


4.0 


4.0 


3.9 


8 


5.5 


5.8 


5.7 


4.8 


8 


4.3 


3.0 


3.4 


3.2 



Differences in raw growth were mixed between Project and non-Project students. These values can be seen in the tables 
above. Project students were mostly lower in Math except grades K, 5, and 8 where Project students outperformed non- 
Project students. In Reading, Project students had smaller raw growth than non-Project students in all grades except 8 th 
where Project students had more raw growth than non-Project student and even more raw growth than NWEA norms. 
Note that the pattern of decreasing raw growth in higher grades is consistent with the national results. 

Comparing RIT scores and raw growth for Project BIE students, non-Project BIE students, and NWEA national norms is 
one way to look at performance, but measuring student growth on an normalized scale is a better method for 
determining how well schools are doing, particularly for schools that have traditionally had scored so far below national 
averages. 

The growth index score shows whether the student grew more than the average of NWEA students who started with the 
same RIT score in the fall term (positive values), or whether the student's growth was below those national norms 
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(negative value). A value of 0 means the student's growth was equal to the average growth of NWEA students who 
started at the same score. The advantage of this comparison over looking at raw growth is that students at different RIT 
score levels have predictably different growth scores (lower grades show more raw growth than upper grades). The 
norm growth index score equalizes growth across all grades by comparing raw growth to the NWEA norms at each grade 
level. Using this measurement results in a more apples-to-apples effect. 

Figure 5: Fall to Spring Math growth index scores 
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Figure 6: Fall to Spring Reading growth index scores 
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Table 7: Fall to Spring Norm-Based Growth Index Scores 



Math Reading 



Fall- 


Grade 


Project Non-Project All BIE 


Norm 




Grade 


Project Non-Project All BIE 


Norm 


K 




K 




1 


1 


2 


-4.12 


-2.53 


-3.16 


0.00 


2 


-5.58 


-4.33 


-4.83 


0.00 


Spring 


3 


-2.38 


-0.21 


-0.97 


0.00 


3 


-5.63 


-2.83 


-3.81 


0.00 


Norm 


4 


-3.73 


-2.27 


-2.83 


0.00 


4 


-2.95 


-1.57 


-2.11 


0.00 


Growth 


5 


-2.70 


-3.36 


-3.12 


0.00 


5 


-2.86 


-0.73 


-1.51 


0.00 




6 


-2.64 


-2.73 


-2.70 


0.00 


6 


-3.19 


-1.28 


-2.02 


0.00 


7 


-2.57 


-2.50 


-2.52 


0.00 


7 


-2.89 


-1.77 


-2.15 


0.00 


8 


-1.36 


-0.79 


-0.97 


0.00 


8 


-1.71 


-1.91 


-1.85 


0.00 


Total 


-2.45 


-1.69 


-1.98 


0.00 


Total 


-2.18 


-1.37 


-1.68 


0.00 



Table 6 shows that the growth index scores of BIE schools were, on average, well below the NWEA norming group in all 
grades. This means that BIE students showed less improvement than the students within the NWEA norming group who 
started with the same score. For instance, for 2 nd grade Math tests, the BIE Project school average was 4.12 points less 
growth than other NWEA students with the same starting score. The BIE non-Project school average was 2.53 points less 
growth, and the virtual comparison group had 2.90 points less. 

The following table shows the same information, but highlights in green where statistical tests of significance indicate 
that non-Project students were outperforming Project students on the Norm Growth Index. 

Table 8: Difference between Project Students and Non-Project Students 





MATH 






READING 




Count 
of Students 


Norm ( 
Inb 


Growth 

ex 




Count 
of Students 


Norm ( 
Ind 


Growth 

ex 


Grade 


Project 


Non- 

Project 


Project 


Non- 

Project 




Project 


Non- 

Project 


Project 


Non- 

Project 


2 


373 


567 


- 4.12 


- 2.53 




372 


569 


- 5.58 


-4.33 


3 


331 


619 


- 2.38 


-.21 




334 


622 


- 5.63 


- 2.83 


4 


375 


606 


- 3.73 


- 2.27 




369 


583 


- 2.95 


- 1.57 


5 


331 


597 


- 2.70 


- 3.36 




333 


572 


- 2.86 


-.73 


6 


352 


561 


- 2.64 


- 2.73 




349 


552 


- 3.19 


- 1.28 


7 


289 


566 


- 2.57 


- 2.50 




296 


570 


- 2.89 


- 1.77 


8 


271 


573 


- 1.36 


-.79 




255 


571 


- 1.71 


- 1.91 


Total Grades 2-8 


2322 


4089 


- 2.87 


- 2.04 




2308 


4039 


- 3.64 


- 2.07 


Total Grades 2-11 


2504 


5469 


- 2.80 


- 2.28 




2496 


5508 


- 3.50 


- 1.82 



In most grades Project students showed lower norm-indexed growth than non-Project students, and in most grades the 
difference was statistically significant. The total for all grades 2-8 showed that non-Project students had Norm-Based 
Growth Index scores statistical higher than Project students. The exceptions were 5 th and 6 th grade Math and 8 th grade 
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Reading where Project students scored better than non-Project students, although this difference was not statistically 
significant. 

VIRTUAL COMPARISON GROUP ANALYSIS 

Virtual Comparison Group (VCG) growth is another way of measuring growth performance for a school or group of 
schools. The VCG score is made up of the average scores for each BIE student's group of comparison students with the 
same starting RIT score, the same school poverty rate, and the same geography. The advantage of this method is that it 
further equalizes the measurement for a particular group of students. For each BIE student, a comparison group of 51 
students who had the same starting RIT score, the same level of school poverty, and the same geographical type (urban 
versus rural) was created. For more information about how these two comparison analyses are created, please see the 
Methodology section of this report and the Appendix. 

The tables below show the average starting test score for BIE students (both Project and non-Project). Next to it shows 
raw growth for the BIE group, the VCG group, and the national normative group. The Norm Growth Index shows how far 
above or below the national norms the BIE group is. The VCG Growth Index shows how far above or below the VCG 
group average the BIE group is. The last set of columns also shows Norm Growth Index and VCG Growth Index for the 
Project schools only. The count of these students has been listed in previous tables and is in the Appendix. 

Table 9: Raw growth for BIE, VCG, and Norms; Norm growth index and VCG growth index for all BIE and for BIE Project 



MATH 


Count of 
Students 


BIE Student 
Fall RIT 




BIE Raw 
Growth 


VCG Raw 
Growth 


Norm Raw 
Growth 




BIE 

Norm Growth 
Index 


VCG Growth 
Index 




BIE Project 
Norm Growth 
Index 


VCG Growth 
Index 


K 


488 


132 




15.3 


16.3 




— 




-1.08 






-0.15 


1 


598 


153 




14.5 


15.7 


-1.07 




-1.46 


2 


995 


172 




11.6 


11.8 


13.2 




-3.16 


-0.25 




-4.12 


-0.53 


3 


994 


183 




10.5 


11.4 


10.7 




-0.97 


-0.86 




-2.38 


-1.73 


4 


1008 


192 




8.9 


8.9 


8.9 




-2.83 


-0.08 




-3.73 


-0.66 


5 


937 


200 




7.1 


8.0 


7.3 




-3.12 


-0.95 




-2.70 


-0.50 


6 


934 


206 




6.2 


7.2 


6.4 




-2.70 


-0.97 




-2.64 


-0.63 


7 


881 


212 




4.9 


5.1 


6.0 




-2.52 


-0.23 




-2.57 


0.34 


8 


869 


217 




5.7 


4.4 


4.8 




-0.97 


1.36 




-1.36 


1.11 


9 


650 


221 




3.4 


2.7 


3.0 




-2.73 


0.74 




10 


541 


225 




1.8 


1.7 


2.7 




-3.04 


0.12 




11 


458 


227 




1.9 


1.5 






-2.81 


0.40 


12 


198 


231 




1.7 


1.3 








0.38 


Total 


9551 












-2.44 


-0.24 




-2.80 


-0.41 
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Table 10: Raw growth for BIE, VCG, and Norms; Norm growth index and VCG growth index for all BIE and BIE Project 



READING 


Count of 
Students 


BIE Student 
Fall RIT 




BIE Raw 
Growth 


VCG Raw 
Growth 


Norm Raw 
Growth 




BIE 

Norm Growth 
Index 


VCG Growth 
Index 




BIE Project 
Norm Growth 
Index 


VCG Growth 
Index 


K 


511 


133 




13.9 


15.7 




- 




-1.78 






-1.86 


1 


625 


151 




12.0 


14.4 


-2.43 




-2.31 


2 


1000 


167 




11.4 


12.8 


12.5 




-4.83 


-1.34 




-5.58 


-1.21 


3 


990 


178 




7.9 


9.5 


8.8 




-3.81 


-1.60 




-5.63 


-3.12 


4 


992 


187 




6.8 


7.4 


6.6 




-2.11 


-0.65 




-2.95 


-1.46 


5 


911 


193 




5.4 


6.1 


5.1 




-1.51 


-0.72 




-2.86 


-2.15 


6 


923 


199 




4.4 


5.4 


4.3 




-2.02 


-1.00 




-3.19 


-2.03 


7 


907 


203 




4.0 


3.7 


3.9 


: 


-2.15 


0.29 


: 


-2.89 


-0.19 


8 


853 


207 




3.4 


3.4 


3.2 




-1.85 


-0.02 




-1.71 


0.02 


9 


675 


211 




2.8 


1.5 


1.9 




-0.47 


1.26 




10 


594 


215 


: 


1.5 


0.3 


1.4 




-1.28 


1.19 


11 


485 


216 




0.7 


0.0 






-2.06 


0.66 


12 


193 


218 




-0.1 


-0.3 








0.24 


Total 


9659 












-2.34 


-0.55 




-3.50 


-1.53| 



Note: VCG Fall RIT is not shown because it is almost identical to BIE students' Fall RIT since the comparison group was 
chosen in part based on their Fall RIT. Because the two groups had the same starting RIT, the difference between their 
raw growth scores is equivalent to the difference between their Spring RITs. 

The differences in the table above show that BIE students lag behind VCG students in most early grades K-7 and 
exceeded VCG growth in upper grades 8-12. In early grades, BIE students lagged behind their comparison groups more in 
Reading than Math. In upper grades, however, BIE students outperformed their VCGs in Math and even more so in 
Reading. For instance, in 9 th and 10 th grade Reading, BIE students averaged over one point of growth more than their 
VCGs. 

Because national norms are not available for Kindergarten and 1 st grade, we can't tell how students in these grades 
compare when their growth is normalized based on NWEA norms. In the total for all grades, BIE students had over two 
points less growth than the norms, but a half point or less difference between BIE and VCG. Again, Project students 
showed less growth in Reading than non-Project students. 

In addition to looking at the differences between the BIE core students and the VCG students, it is also important to look 
at whether those differences are statistically significant. Statistical significance is a measure of whether the difference 
between two groups of students is different from random chance. A confidence level of p=.05 was used. 

The table below shows the difference in Raw Growth between BIE students and VCG students. The dark green 
highlighting shows areas where BIE core students were statistically significantly higher than the VCG group, and light red 
highlighting shows areas where BIE core students were statistically significantly lower than the VCG group. 
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Table 11: Difference between BIE students and their VCG group 

VCG students have the same starting score, poverty level, and geography 





BIE Raw 
Growth 


VCG Raw 
Growth 


Raw 

Growth 

Difference 


K 


15.3 


16.3 


-1.1 


1 


14.5 


15.7 


-1.2 


2 


11.6 


11.8 


-0.3 


3 


10.5 


11.4 


-0.9 


4 


8.9 


8.9 


-0.1 


5 


7.1 


8.0 


-0.9 


6 


6.2 


7.2 


-1.0 


7 


4.9 


5.1 


-0.2 


8 


5.7 


4.4 


1.4 


9 


3.4 


2.7 


0.8 


10 


1.8 


1.7 


0.1 


11 


1.9 


1.5 


0.4 


12 


1.7 


1.3 


0.4 



Reading 





BIE Raw 
Growth 


VCG Raw 
Growth 


Raw 

Growth 

Difference 


K 


13.9 


15.7 


-1.8 


1 


12.0 


14.4 


-2.4 


2 


11.4 


12.8 


-1.4 


3 


7.9 


9.5 


-1.6 


4 


6.8 


7.4 


-0.7 


5 


5.4 


6.1 


-0.7 


6 


4.4 


5.4 


-1.0 


7 


4.0 


3.7 


0.3 


8 


3.4 


3.4 


0.0 


9 


2.8 


1.5 


1.3 


10 


1.5 


0.3 


1.2 


11 


0.7 


0.0 


0.7 


12 


-0.1 


-0.3 


0.2 



highlighting shows statistical significance 



As can be seen in the table above, BIE students did not have as much raw growth as their VCG groups in grades K-6, 
although two grade level differences in Math were not statistically significant. In the upper grades, BIE students showed 
better raw growth than their VCG groups; this difference was statistically significant for Math in grades 8 and 9, and for 
Reading in grades 9 and 10. The following table shows that the same general pattern applies to Project students, 
although grades 9-12 cannot be shown due to insufficient sample size. 
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Table 12: Difference between BIE Project students and their VCG group 

VCG students have the same starting score, poverty level, and geography 





Math 




Reading 






BIE Raw 
Growth 


VCG Raw 
Growth 


Raw 

Growth 

Difference 






BIE Raw 
Growth 


VCG Raw 
Growth 


Raw 

Growth 

Difference 


K 


15.7 


15.8 


-0.1 




K 


13.3 


15.2 


-1.9 




13.6 


15.2 








11.8 


14.1 




1 






-1.6 




1 






-2.3 


2 


10.8 


11.4 


-0.5 




2 


11.1 


12.4 


-1.3 


3 


9.2 


11.0 


-1.7 




3 


6.6 


9.7 


-3.1 


4 


8.3 


8.9 


-0.7 




4 


6.5 


8.0 


-1.5 


5 


7.6 


8.1 


-0.5 




5 


4.8 


6.9 


-2.1 


6 


6.3 


6.9 


-0.6 




6 


3.7 


5.7 


-2.0 


7 


5.0 


4.5 


0.5 




7 


3.9 


4.1 


-0.2 


8 


5.5 


4.4 


1.1 




8 


4.3 


4.3 


0.0 



highlighting shows statistical significance 



DIFFERENCE FOR TYPES OF SCHOOLS 
Difference for Professional Development 

A number of the schools in the BIE have utilized the services of the NWEA professional development team to teach, 
train, and coach teachers and school leaders in using MAP data to make informed decisions for students. The NWEA 
professional development services received ranged in nature: some schools may have had several full days of whole- 
school workshops or work with school leadership teams, while other schools may have been engaged on a more limited 
basis. Although there is a large degree of difference in the professional development services received, we wanted to 
explore whether this service in aggregate showed an effect. 

The following tables show the calculated difference between students from schools that had any Professional 
Development time with NWEA (PD) and schools that didn't (No PD). In addition, tests were run to see if there was any 
statistically significant difference between the two groups. Areas where students from PD schools scored statistically 
significantly (p < .05) higher on any metric are highlighted in green. Areas where students from PD schools scored 
statistically significantly lower than students from schools without NWEA PD are highlighted in pink. Fall RIT score, 
Spring RIT, Norm Index Score, and VCG Index Score are all shown. The VCG Growth Index Score for the PD group is 
comprised of a comparison group of students matched to students in schools with PD; students in their VCG group may 
or may not have received professional development services. The comparison being made is between students in 
schools which had PD services compared to the overall comparison group for those students. Likewise, the VCG Growth 
Index for the No PD group reflects national NWEA students who may or may not have received PD. 
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Table 13: Math Fall, Spring, Norm Growth Index, and VCG Growth Index for Professional Development groups 



Math 


Count 




Fall09 RIT 


SpringlORIT 


Norm Growth 
Index 


VCG Growth 
Index 




PD 


No PD 




PD 


No PD 


PD 


No PD 




PD 


No PD 


PD 


No PD 


K 


373 


115 




132.8 


130.4 


146.7 


144.6 








-0.51 


-2.93 


1 


442 


156 




154.7 


149.4 


166.2 


164.6 








-1.13 


-0.91 


2 


489 


506 




171.2 


172.2 


182.8 


183.4 




-3.12 


-3.21 


0.14 


-0.66 


3 


453 


541 




180.7 


184.1 


191.5 


194.5 




-1.17 


-0.80 


-0.88 


-0.83 


4 


505 


503 




190.6 


194.0 


200.1 


202.1 




-2.52 


-3.15 


0.39 


-0.57 


5 


440 


497 




198.5 


201.7 


207.0 


207.7 




-1.83 


-4.28 


0.45 


-2.19 


6 


441 


493 




204.5 


208.3 


211.2 


214.4 




-2.59 


-2.79 


-0.69 


-1.23 


7 


364 


517 




209.0 


213.6 


214.1 


218.5 




-2.16 


-2.77 


0.60 


-0.80 


8 


306 


563 




215.5 


217.9 


221.3 


223.3 




-0.89 


-1.01 


1.70 


1.17 


9 


112 


538 




225.0 


220.7 


228.0 


224.1 




-2.81 


-2.72 


1.06 


0.67 


10 


110 


431 




226.2 


225.0 


229.2 


226.8 




-1.72 


-3.39 


1.45 


-0.23 


11 


95 


363 




225.1 


227.7 


229.4 


229.1 




-0.69 


-3.37 


2.72 


-0.21 


12 


87 


111 




231.9 


230.4 


233.1 


232.7 






-0.07 


0.74 


Total 


4217 


5334 






-2.09 


-2.68 


0.10 


-0.52 



Table 14: Reading Fall, Spring, Norm Growth Index, VCG Growth Index for Professional Development groups 



Reading 


N 




Fall09 RIT 


SpringlO RIT 


Norm Growth 
Index 


VCG Growth 
Index 




PD 


No PD 




PD 


No PD 


PD 


No PD 




PD 


No PD 


PD 


No PD 


K 


386 


125 


134.3 


130.7 


142.6 


146.4 




-1.65 


-2.15 


1 


455 


170 


152.3 


147.1 


165.0 


159.6 


-2.04 


-3.54 


2 


488 


512 


165.5 


168.4 


177.2 


179.9 


-5.08 


-4.6 


-1.11 


-1.56 


3 


460 


530 


175.3 


180.2 


183.1 


188.3 


-4.60 


-3.1 


-2.33 


-0.96 


4 


501 


491 


185.0 


189.1 


192.0 


195.7 


-2.28 


-1.9 


-0.78 


-0.52 


5 


444 


467 


190.5 


196.3 


196.1 


201.6 


-1.83 


-1.2 


-1.09 


-0.37 


6 


437 


486 


195.8 


201.0 


200.1 


205.5 


-2.65 


-1.5 


-1.65 


-0.41 


7 


383 


524 


199.8 


205.8 


203.8 


209.6 


-2.65 


-1.8 


-0.04 


0.53 


8 


302 


551 


203.2 


208.9 


207.5 


212.0 


-1.74 


-1.9 


0.03 


-0.05 


9 


128 


547 


212.5 


211.1 


215.7 


213.6 


-0.16 


-0.5 


1.67 


1.17 


10 


129 


465 


214.9 


214.9 


216.7 


216.3 


-1.81 


-1.1 


0.73 


1.31 


11 


104 


381 


213.9 


216.2 


215.9 


216.6 


-1.15 


-2.3 


1.44 


0.46 


12 


87 


106 


218.1 


217.4 


217.9 


217.4 




0.07 


0.39 


Total 


4304 


5355 






-2.87 


-1.98 


-1.01 


-0.18 
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In Math, students from schools with NWEA Professional Development courses started at lower fall scores in grades 3-8 
and higher fall grades in grades 1 and 9. Norm-based growth index scores were higher for the PD group in total and in 
grades 5 and 11. VCG index scores for Math were higher overall and in grades K, 7 and 11. In grade 11, for instance, 
students from schools with NWEA PD scored 2.72 points above their VCG while students from schools without PD scored 
.21 below their VCG. In Reading, students in schools with Professional Development started at lower scores and stayed 
at lower scores in grades 2-8, and showed less norm-based and VCG-based growth in total and in grades 3 and 6 than 
students from schools without PD. For 1 st grade Reading, however, better scores and better growth was seen for 
students from schools with NWEA PD courses. The difference in results between Reading and Math is pronounced; we 
recommend that the BIE look at curriculum and instructional practices to see what may have influenced this difference. 

Note that schools may have received professional development in any grade; grades that did not show a difference for 
PD may be because that grade did not receive PD. More information is needed about the nature of the professional 
development received before true inferences can be drawn. Note also that norm growth information is not available for 
Kindergarten and 1 st grade where VCG growth showed a difference for PD schools. 



Difference for Tenure with NWEA 

The following tables show the calculated difference between students from schools that had first tested with NWEA 
before the 2008-2009 school year (>2 years) and schools that started testing in the last two years (new). Statistical 
significance is shown as it was for the Professional Development chart: differences in growth favoring schools with long 
tenure are in green, differences favoring the new schools are in pink. Note that "new school" does not mean that the 
school was new; it means that the school is a relative newcomer to NWEA testing. As with the previous set of tables, the 
VCG Growth Index shows the difference between BIE students in schools that have been testing with NWEA for more 
than 2 years compared to a national group that may or may not have been testing for 2 years. 

Table 15: Math Fall, Spring, Norm Growth Index, and VCG Growth Index for NWEA tenure groups 



Math 


Count 




Fal 109 RIT 


SpringlO RIT 


Norm Growth 
Index 


VCG Growth 
Index 




>2 

years 


new 




>2 

years 


new 


>2 

years 


new 




>2 

years 


new 


>2 

years 


new 


K 


384 


104 




131.6 


134.7 


148.3 


138.3 








-1.13 


-0.89 


1 


427 


171 




151.5 


157.7 


165.1 


167.7 








0.15 


-5.09 


2 


742 


253 




171.3 


172.7 


183.8 


181.2 




-2.27 


-6.23 


0.33 


-2.28 


3 


739 


255 




182.6 


182.4 


193.8 


191.1 




-0.27 


-3.08 


-0.55 


-1.77 


4 


749 


259 




192.3 


192.2 


201.6 


199.7 




-2.38 


-4.21 


0.08 


-0.58 


5 


694 


243 




200.2 


200.1 


207.3 


207.4 




-3.13 


-3.11 


-1.25 


-0.07 


6 


679 


255 




206.2 


207.3 


213.1 


212.2 




-2.03 


-4.55 


-0.59 


-2.05 


7 


613 


268 




212.4 


210.1 


217.6 


214.4 




-2.22 


-3.24 


-0.04 


-0.69 


8 


544 


325 




217.1 


216.8 


223.2 


221.6 




-0.62 


-1.57 


1.76 


0.67 


9 


525 


125 




221.4 


221.7 


224.9 


224.6 




-2.65 


-3.09 


0.79 


0.53 


10 


400 


141 




225.1 


225.8 


226.6 


229.4 




-3.34 


-2.20 


-0.25 


1.17 


11 


323 


135 




227.2 


226.9 


228.3 


231.3 




-3.83 


-0.44 


-0.61 


2.73 


12 


190 


8 




















Total 


7009 


2542 




-2.14 


-3.28 


-0.09 


-0.69 
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Table 16: Reading Fall, Spring, Norm Growth Index, VCG Growth Index for NWEA tenure groups 



Reading 



N 



Fal 109 RIT SpringlORIT 



Norm Growth 
Index 



VCG Growth 
Index 





>2 

years 


new 




>2 

years 


new 


>2 

years 


new 




>2 

years 


new 


>2 

years 


new 


K 


395 


116 




132.0 


138.0 


147.8 


129.3 








-1.97 


-1.02 


1 


456 


169 




149.6 


154.4 


162.8 


165.4 








-2.24 


-3.11 


2 


743 


257 




166.3 


168.9 


179.0 


177.4 




-3.86 


-8.13 


-1.00 


-2.49 


3 


737 


253 




177.9 


177.9 


186.2 


184.9 




-3.45 


-4.94 


-1.52 


-1.86 


4 


738 


254 




186.5 


188.4 


194.1 


192.8 




-1.41 


-4.19 


-0.27 


-1.79 


5 


669 


242 




192.8 


195.2 


199.1 


198.5 




-0.77 


-3.56 


-0.19 


-2.18 


6 


673 


250 




197.9 


200.5 


203.1 


202.5 




-1.26 


-4.17 


-0.40 


-2.70 


7 


628 


279 




203.4 


202.9 


207.4 


206.6 




-2.00 


-2.49 


0.41 


0.02 


8 


535 


318 




206.4 


207.8 


210.2 


210.8 




-1.73 


-2.05 


-0.01 


-0.03 


9 


542 


133 




211.1 


212.7 


213.1 


217.5 




-0.98 


1.53 


0.74 


3.25 


10 


443 


151 




213.7 


218.4 


215.2 


220.0 




-1.42 


-0.89 


0.80 


2.27 


11 


353 


132 




214.8 


218.1 


215.4 


219.4 




-2.37 


-1.28 


0.13 


2.03 


12 


185 


8 




















Total 


7097 


2562 




-1.98 


-3.32 


-0.45 


-0.85 



The effect of splitting the data by length of time testing with NWEA is dramatic. For both Math and Reading, students 
from schools testing with NWEA for more than 2 years showed better growth, relative to both the norms and their VCG, 
than schools testing less than two years. In some cases, the differences were very large. For instance in Math, 2 nd grade 
students were behind the norm growth by 2.27 points while students from schools that were newer with NWEA lagged 
by 6.23 points. In the 5 th grade in Reading, long time testers were almost at the norm growth (-.77) while newer testers 
lagged by over three and a half points. 

The upper grades show a different pattern. There was no statistically significant difference between the two groups for 
most grades 7-11 in both subjects, except for 11 th grade math and 9 th grade Reading where students from long tenure 
schools had lower growth and 11 th grade for the VCG Index. One possible explanation for this is that testing doesn't 
begin for all grades at the same time. Many schools working with NWEA start by testing the early grades, then add high 
school testing later. No information was available to determine if testing time by grade had a similar effect for upper 
grades as it did for lower grades. Note that the 12 th grade cannot be measured because the sample size is too small. 



Difference for BIE School Type 

The following tables show the calculated difference between students from schools operated by the Bureau (BIE Day 
Schools and BIE Boarding Schools) and schools operated through partners (Contract Day Schools, Grant Day Schools, and 
Grant Boarding Schools). Statistical significance is shown as it was for the prior two sections: differences where Bureau- 
operated student performance was higher than non-Bureau-operated school performance are highlighted in green; 
where Bureau performance was lower are highlighted in pink. As with the previous set of tables, the VCG Growth Index 
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shows the difference between BIE students in Bureau schools compared to a national group; none of the VCG group is in 
either Bureau or BIE non-Bureau schools. 

Table 17: Math Fall, Spring, Norm Growth Index, and VCG Growth Index for BIE school type 



Math 



Count 



Fal 109 RIT 



SpringlO RIT 



Norm Growth 
Index 



VCG Growth 
Index 





Bureau 


Non- 

Bureau 




Bureau 


Non- 

Bureau 


Bureau 


Non- 

Bureau 




Bureau 


Non- 

Bureau 


Bureau 


Non- 

Bureau 


K 


232 


256 




132.0 


132.5 


150.0 


142.7 








0.09 


-2.21 


1 


283 


315 




152.7 


153.8 


164.7 


166.8 








0.52 


-2.32 


2 


389 


606 




172.2 


171.3 


184.7 


182.1 




-2.04 


-3.93 


0.53 


-0.79 


3 


396 


598 




183.4 


182.0 


195.0 


191.8 




0.56 


-2.02 


0.46 


-1.76 


4 


414 


594 




192.0 


192.5 


202.2 


200.4 




-1.27 


-3.92 


1.39 


-1.11 


5 


386 


551 




199.7 


200.6 


207.8 


207.1 




-2.14 


-3.81 


-0.19 


-1.48 


6 


384 


550 




207.1 


206.0 


214.2 


212.0 




-1.91 


-3.26 


-0.24 


-1.49 


7 


288 


593 




213.4 


210.9 


219.5 


215.3 




-1.37 


-3.09 


0.84 


-0.75 


8 


268 


601 




219.0 


216.1 


225.5 


221.3 




0.45 


-1.57 


2.86 


0.74 


9 


247 


403 




220.4 


222.1 


225.7 


224.3 




-1.06 


-3.77 


2.25 


-0.20 


10 


140 


401 




221.8 


226.5 


224.6 


228.3 




-2.42 


-3.27 


0.21 


0.09 


11 


141 


317 




224.7 


228.2 


226.7 


230.3 




-2.97 


-2.73 


0.06 


0.56 


12 


119 


79 




229.2 


233.9 


231.5 


235.0 






0.85 


-0.30 


Total 


3687 


5864 






-1.30 


-3.12 


0.72 


-0.85 



Table 18: Reading Fall, Spring, Norm Growth Index, VCG Growth Index for BIE school type 



Reading 



N 



Fal 109 RIT SpringlO RIT 



Norm Growth 
Index 



VCG Growth 
Index 





Bureau 


Non- 

Bureau 




Bureau 


Non- 

Bureau 


Bureau 


Non- 

Bureau 




Bureau 


Non- 

Bureau 


Bureau 


Non- 

Bureau 


K 


236 


275 




131.5 


135.0 


149.7 


138.3 








-0.30 


-3.15 


1 


298 


327 




149.7 


152.0 


164.4 


162.7 








-1.58 


-3.13 


2 


386 


614 




166.6 


167.3 


180.2 


177.6 




-2.80 


-6.20 


0.28 


-2.43 


3 


397 


593 




177.5 


178.2 


186.4 


185.5 




-2.83 


-4.48 


-1.29 


-1.81 


4 


412 


580 




184.8 


188.6 


193.6 


193.9 




-0.33 


-3.42 


0.83 


-1.75 


5 


388 


523 




192.1 


194.5 


198.1 


199.5 




-1.05 


-1.85 


-0.54 


-0.86 


6 


384 


539 




197.2 


199.6 


202.0 


203.7 




-1.62 


-2.31 


-1.02 


-0.99 


7 


289 


618 




201.7 


204.0 


206.0 


207.7 




-2.06 


-2.19 


0.00 


0.44 


8 


259 


594 




204.4 


208.0 


209.6 


210.8 




-0.64 


-2.37 


0.96 


-0.45 


9 


244 


431 




209.3 


212.6 


214.2 


213.9 




1.62 


-1.73 


3.29 


0.03 


10 


140 


454 




209.7 


216.5 


212.0 


217.8 




-0.62 


-1.48 


1.02 


1.24 


11 


145 


340 




211.1 


217.7 


213.0 


217.9 




-1.44 


-2.33 


0.64 


0.68 


12 


118 


75 




215.0 


222.1 


214.8 


222.1 






-0.33 


1.12 


Total 


3696 


5963 






-1.33 


-2.94 


0.03 


-0.91 
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Disaggregation by Bureau and non-Bureau schools shows a clear pattern. Bureau schools started at performance levels 
that are similar to non-Bureau schools in fall 2009. Spring results indicate that Bureau schools showed significantly 
greater growth than non-Bureau schools (when compared to both the norms and their respective Virtual Comparison 
Groups) and the positive differences in growth led to differences in performance that favored students attending the 
Bureau schools. This was true in both Mathematics and Reading. 



Difference for ADD, ELO, and State 

Because there were a large number of ADDs, ELOs, and states, analyses for these groupings have been limited to the 
VCG Growth Index and the Norm Growth Index for all grades combined. For each grouping, the results have been listed 
in order from highest to lowest Fall to Spring norm-indexed growth. Note that the order from highest to lowest may be 
different for VCG difference. The total for all students is listed and highlighted in blue to show states, ADDs, and ELOs 
which have better growth scores above the dark total line. 



Table 19: Math Norm Growth Index and VCG Growth Index for ELOs 



ELO 


Math average across all grades 


Count for 
VCG Growth 


Count for 
Norm 
Growth 




VCG Growth 
Index 


Norm 

Growth 

Index 


Standing Rock ELO 


104 


84 




3.87 


4.21 


New Mexico Navajo North ELO 


949 


702 




2.90 


1.62 


New Mexico North ELO 


402 


402 




1.24 


-1.66 


Minneapolis ELO 


1015 


905 




-0.18 


-2.02 


Turtle Mountain ELO 


1116 


1066 




-0.42 


-2.17 


Seattle ELO 


281 


243 




0.67 


-2.19 


New Mexico Navajo Central ELO 


1098 


894 




-0.50 


-2.20 


New Mexico South ELO 


231 


231 




1.67 


-2.23 


Arizona Navajo North ELO 


565 


542 




-0.28 


-2.43 


Total 


9093 


7973 




-0.24 


-2.44 


Sacramento ELO 


50 


50 




0.24 


-2.86 


Southern & Eastern States ELO 


126 


126 




-1.21 


-2.98 


Arizona Navajo Central ELO 


1007 


841 




-1.49 


-4.04 


New Mexico Navajo South ELO 


1511 


1354 




-1.17 


-4.04 


Arizona Navajo South ELO 


123 


105 




-0.59 


-4.36 


Arizona South ELO 


292 


248 




-3.86 


-4.53 


Billings ELO 


164 


140 




-1.84 


-4.60 


Oklahoma ELO 


59 


40 




-2.76 


-5.10 
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Table 20: Reading Norm Growth Index and VCG Growth Index for ELOs 





Reading average across all grades 


ELO 


Count for 
VCG Growth 


Count for 
Norm 




VCG Growth 
Index 


Norm 

Growth 




Growth 




Index 


Standing Rock ELO 


105 


85 




1.96 


1.41 


New Mexico Navajo North ELO 


953 


706 




2.12 


1.01 


New Mexico North ELO 


422 


422 




2.72 


0.07 


Minneapolis ELO 


1018 


912 




0.46 


-1.25 


Sacramento ELO 


55 


55 




1.22 


-1.40 


Turtle Mountain ELO 


1176 


1133 




0.17 


-1.58 


Seattle ELO 


231 


185 




0.12 


-1.78 


Total 


9185 


8004 




-0.55 


-2.34 


New Mexico South ELO 


229 


229 




0.78 


-2.60 


Arizona Navajo North ELO 


555 


535 




-0.98 


-2.89 


New Mexico Navajo Central ELO 


1099 


886 




-1.80 


-3.12 


New Mexico Navajo South ELO 


1512 


1353 




-1.21 


-3.13 


Southern & Eastern States ELO 


125 


125 




-2.02 


-3.56 


Billings ELO 


184 


161 




-2.36 


-4.31 


Arizona Navajo South ELO 


131 


109 




-1.05 


-4.61 


Arizona Navajo Central ELO 


1033 


822 




-2.99 


-4.84 


Arizona South ELO 


295 


244 




-3.31 


-4.92 


Oklahoma ELO 


62 


42 




-1.65 


-7.21 



Table 21: Math Norm Growth Index and VCG Growth Index for ADDs 



ADD 


Math average across all grades 


Count for 
VCG Growth 


Count for 
Norm 
Growth 




VCG Growth 
Index 


Norm 

Growth 

Index 


ADD East 


1405 


1316 




-0.27 


-1.93 


Total 


9093 


7973 




-0.24 


-2.44 


ADD West 


2435 


2219 




-0.22 


-2.46 


ADD Navajo 


5253 


4438 




-0.25 


-2.58 



Table 22: Reading Norm Growth Index and VCG Growth Index for ADDs 



ADD 


Reading average across all grades 


Count for 
VCG Growth 


Count for 
Norm 
Growth 




VCG Growth 
Index 


Norm 

Growth 

Index 


ADD East 


1468 


1385 




0.03 


-1.75 


ADD West 


2434 


2208 




0.20 


-1.81 


Total 


9185 


8004 




-0.55 


-2.34 


ADD Navajo 


5283 


4411 




-1.05 


-2.79 



28 | P a g e 







Table 23: Math Norm Growth Index and VCG Growth Index for states 



State 


Math average across all grades 


Count for 
VCG Growth 


Count for 
Norm 
Growth 




VCG Growth 
Index 


Norm 

Growth 

Index 


UT 


137 


103 




2.58 


0.47 


OR 


141 


105 




2.93 


0.19 


Ml 


141 


102 




0.50 


-0.49 


MN 


243 


189 




-0.24 


-1.08 


ND 


1297 


1227 




0.04 


-1.60 


Wl 


377 


376 




-0.76 


-2.39 


Total 


9093 


7973 






-2.44 


WY 


108 


86 




-0.10 


-2.51 


NM 


3664 


3178 




-0.08 


-2.63 


AZ 


2377 


2038 




-0.68 


-2.69 


NV 


50 


50 




0.24 


-2.86 


ME 


126 


126 




-1.21 


-2.98 


WA 


140 


138 




-1.61 


-4.00 


IA 


177 


161 




-0.16 


-4.18 


KS 


59 


40 




-2.76 


-5.10 


MT 


56 


54 




-5.20 


-7.93 



Table 24: Reading Norm Growth Index and VCG Growth Index for states 



State 


Reading average across all grades 


Count for 
VCG Growth 


Count for 
Norm 
Growth 




VCG Growth 
Index 


Norm 

Growth 

Index 


OR 


141 


103 




1.40 


0.02 


Ml 


142 


103 




-0.15 


-0.56 


UT 


134 


102 




1.16 


-0.65 


Wl 


363 


362 




0.88 


-0.89 


ND 


1358 


1295 




0.38 


-1.27 


NV 


55 


55 




1.22 


-1.40 


MN 


257 


208 




-0.47 


-1.93 


NM 


3691 


3191 




-0.37 


-2.27 


IA 


179 


162 




0.97 


-2.33 


Total 


9185 


8004 




-0.55 


-2.34 


WY 


123 


102 




-1.28 


-2.87 


AZ 


2404 


2013 




-1.67 


-3.36 


ME 


125 


125 




-2.02 


-3.56 


WA 


90 


82 




-1.88 


-4.05 


MT 


61 


59 




-4.54 


-6.80 


KS 


62 


42 




-1.65 


-7.21 
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Separating norm-indexed growth scores and VCG differences by ADD, ELO, and State showed interesting results. Two 
ELOs stood out as achieving more growth in comparison to other ELOs: in Math, Standing Rock ELO had an average of 
4.21 growth points above the national norms, and New Mexico Navajo North had 1.62 growth points above. This is in 
comparison to an average across all ELOs of -2.44 points. Standing Rock and New Mexico Navajo North, along with New 
Mexico North and Minneapolis also had higher growth rates in comparison to their VCG groups. 

In Math, ADD East had better norm growth index scores than the other two ADDs, but all ADDs scored similarly on VCG 
index. This suggests that while ADD East has better results than the other ADDs in growth, their schools were comprised 
of different poverty and geography characteristics than the other two. In Reading, ADDs East and West scored 
significantly better than ADD Navajo on both the Norm Index Growth Score and the VCG Growth Index Score. 

Among states, Oregon and Utah showed higher growth scores than the national NWEA norms while other states score 
below. Oregon and Utah were also significantly higher than their VCG groups. For instance, Oregon scored almost three 
points higher than their VCG group. In Reading, Oregon was still the highest performer on the Norm Growth Index, 
although it was exactly at the norm, not above. Several states showed better growth than their VCG groups: Oregon, 
Nevada, Utah, Iowa, Wisconsin, and North Dakota. 

Note that these comparisons have not been separated by grade level. While norm-indexed growth scores are 
comparable across grades, there may be differences between the grades. ADDs, ELOs, and states with higher growth 
scores overall may not have higher growth on a grade-by-grade basis. 
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DISCUSSION 

MEASURING PERFORMANCE FOR BIE STUDENTS 

The main way this report is different from other types of performance reports is that it measures actual student growth, 
not just whether student performed above a particular state's cut score. In addition, MAP tests measure the level of 
subject matter knowledge in reading and math that a student has attained, not simply whether they have met a grade- 
level standard. BIE students have often not passed state AYP measures and are mislabeled with the term "failing." 
Achievement for the educational system should be measured on student growth, particularly for students who are 
behind others, so successes can be measured, encouraged, and increased. The Bureau of Indian Education has 
recognized this fact in its partnership with NWEA to help schools and students to understand what it takes for students 
to improve and to see what it looks like when they do. This Project is early in its inception, but the rigorous 
measurement of student performance will, over time, allow the BIE to identify successes as well as challenge areas that 
need more attention. 

SUMMARY OF FINDINGS 

The average starting Fall RIT scores for students in BIE Project schools were lower than non-Project school students, 
possibly because schools were chosen for the Project based on student performance. The average for BIE students— 
Project, non-Project, and overall— was lower than the national norms provided by NWEA. BIE students lagged more in 
Reading than in Math. 

There were two main growth comparisons used in this analysis. The Norm Indexed Growth Score shows whether a group 
of students is above or below the national growth rate for students with the same starting score. The VCG Indexed 
Growth Score shows whether a group of students is above or below the growth rate for a group of students similar to 
them in starting score, school poverty rate, and geographical location (urban or rural). The Norm Index puts all students 
on an equal measurement with students in other grades and states, but the VCG Index further refines the comparison to 
schools that face similar challenges to BIE schools. 

Average BIE school Fall to Spring growth scores were lower than the national norm sample, as indicated by the negative 
Norm Growth index Score. BIE students averaged across all grades had similar performance to their VCG groups. By 
grade, BIE students were generally lower than their VCG group in early grades K-7 and higher than their VCG group in 
later grades 8-12. 2 nd and 8 th grade Math and 9 th grade Reading showed particularly high growth for BIE students 
compared to their comparison groups. The same pattern was true for students in the BIE Project. 

Statistical significance tests show which differences truly reflected a difference between student groups and weren't just 
due to random chance. This is particularly important to measure when only one year of growth data is available and a 
trend cannot be established. Unfortunately, statistical significance also depends on the number of students in each 
group; with a small number of students, differences cannot be identified with certainty. The pattern for these tests of 
significance was generally consistent: BIE students showed less growth than their virtual comparison groups in Math for 
the total across all grades and specifically for grades, grades 3, 5, 6 , and also in Reading grades 3-6. BIE students showed 
statistically more growth than their comparison groups in Math in 8 th and 9 th grade, and in Reading in 9 th and 10 th grade. 
Some of these differences, while showing a strong statistical significance, may not result in a meaningful difference. For 
instance, the difference between the BIE student average and the comparison group average in 4 th grade Math was .65 
norm indexed growth points. A half of a RIT point less growth in the 4 th grade will put the student at a significant 
disadvantage if sustained over several years, but can also be recovered within a school year. 



31 | P a g e 




When disaggregated by type of school, some patterns emerged. Students who were in schools that had NWEA 
Professional Development courses had higher growth rates as measured by Norm Indexed Growth Score and VCG 
Growth Score in Math overall. Growth rates were significantly better in grades K, 7, and 11. In addition, students from 
schools that had been testing with NWEA for more than 2 years showed better growth rates overall and especially in 
grades 1-6. 

When disaggregated by geography, other differences became apparent. ADD East showed better student growth than 
the other ADDs in both subjects, and ADD West also showed good growth in Reading. Standing Rock ELO and New 
Mexico Navajo North ELO had higher growth scores on both metrics in both subjects. New Mexico North and South ELOs 
had better growth than their comparison groups in both subjects, and four more ELOs showed good growth in Reading 
compared to their comparison groups: Minneapolis, Sacramento, Turtle Mountain, and Seattle ELOs were added to this 
list of performers. Among states, Oregon and Utah showed better performance than other states. 

One additional way to separate schools is by BIE status. Bureau-managed schools showed better growth than schools 
that were managed through a grant or contract. This was true for Reading and Math in most grades. In future years 
when there are a larger number of students whose growth can be measured, we recommend doing two separate 
analyses: one for all BIE students, and one for just students in Bureau schools. 

HIGHLIGHTS 

Overall, the academic performance of BIE students reflects the challenges that the group faces in their lives. BIE students 
on the whole have lower test scores and lower growth than a national norm population of students. In addition, BIE 
students in some subjects, grades, and geographic areas have lower growth than a group of students with similar test 
scores, similar school poverty rates, and similar rural geography. The data from this report reflects the difficult situation 
for BIE students and the necessary challenge for BIE schools to overcome. Within the data, however, there were several 
areas with positive outcomes that can be highlighted and should be celebrated and continued. Looking at high-level 
averages often masks the remarkable success stories for some students and schools. A few examples are listed below. 

• While the overall averages show BIE students with less growth than their comparison groups in early grades, 
there were more BIE students who outperformed their comparison groups than who underperformed. In some 
cases, students had vastly more growth than their comparison groups. In 10% of cases, BIE students had at least 
10 points more of raw growth than their comparison groups. 

• Some schools showed phenomenal growth. In one school, the average difference between their students and 
the comparison group was +5.3 points. That means students in that school outperformed comparison students 
by more than a year's worth of growth. In another school, the average growth for its students was +4.9 points 
above the national norms for students with similar scores, again showing remarkable growth. 8 schools had 
average norm-based growth index scores above a full point, meaning they were substantially outperforming the 
norms. 17 schools had average student VCG index scores of more than a point, meaning they were substantially 
outperforming their comparison groups. 

• Although this report does not analyze AYP performance, estimates were made for whether students were 
meeting AYP in each of their states based on their MAP scores. Some ELOs showed an average of 45, 55, even 
60% of their students estimated to meet their state's benchmark. In one school, for example, 77% of students 
were estimated to meet benchmark, a level that is perhaps even above their state average. 
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RECOMMENDATIONS 



After analyzing the performance data for the 2009-2010 school year, we have the following recommendations for the 
Bureau of Indian Education. 

• We recommend that the Bureau continue to look closely at its student performance data on a regular basis. 

NWEA provides achievement and growth reports to BIE after each term. These interactive reports provide Bureau 
leaders as well as ADD, ELO, and school leaders a way to look at student data to find areas of success and areas of 
concern. This printed report shows at a high level how BIE students are doing in a variety of areas, but doesn't take 
the place of teachers and students looking at their data together throughout the year to create pathways for 
progress. 

• We recommend that the Bureau undertake further analysis in areas not pursued in this report. 

There are areas of analysis not included in this report or included only minimally that could provide additional 
information. For instance, disaggregation by schools receiving professional development showed a difference but 
the types of professional development received were studied. More analysis would be needed to understand 
whether the particular type of professional development has an effect for students. In addition, an area of research 
not included in this report is summer loss: whether growth gained by students during the year is sustained over the 
summer months into the next school year. Research in the field of education has shown that many students in 
poverty situations have more loss over the summer than students in affluent situations, meaning that these 
students stay further behind despite significant school year growth. 

• We recommend that the Bureau continue to monitor progress on an annual basis for several years, and then 
commission a program evaluation study including at least three years of student performance. 

This report serves as the first in what we recommend is a series of reports on the progress of BIE schools. With one 
year of testing data, some differences were able to be identified, but more years of data deepens the analyses and 
allows for the identification of trends and progress. Intervention strategies often take several years for results to be 
seen at a high level, so a multi-year evaluation is necessary to reflect the result of Bureau changes. NWEA can work 
with the BIE to identify an appropriate organization to provide such an evaluation. 

• We recommend that the Bureau use their data to ask difficult questions about student performance and 
school programs. 

This report and the "roll up reporting" provided every term gives BIE an opportunity to ask questions about whether 
their current interventions are working, and where they might need more interventions. For instance, in general 
growth for BIE students was better in Math than Reading— was this due to a particular intervention? Could students 
benefit from the same type of intervention in Reading? Why do students in different ELOs show such different 
growth? Are there best practices that could be shared among BIE schools? What is driving the strong growth for high 
school students? 
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• We recommend that the Bureau celebrate and build on successes. 



While some of the average numbers in this report are discouraging, there are examples of exemplary performance 
and growth for students and schools. Finding, celebrating, and replicating successes like these will help encourage 
schools to see what's possible. 



We hope that the data from this report helps the Bureau of Indian Education understand the current state of their 
student achievement. However, data is merely the beginning of process improvement. Answering the questions raised 
by the data and implementing necessary changes is what brings about a true increase in student achievement. 
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APPENDIX A: Counts of Students by Grade and School Group 



MATH 

State 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


AZ 


143 


226 


303 


295 


288 


280 


263 


236 


212 


141 


45 


31 


13 


2476 


IA 


0 


17 


21 


17 


12 


13 


22 


22 


14 


15 


17 


10 


0 


180 


KS 


9 


3 


4 


6 


7 


2 


3 


3 


1 


2 


5 


7 


7 


59 


ME 


0 


0 


12 


12 


17 


25 


22 


19 


19 


0 


0 


0 


0 


126 


Ml 


24 


12 


15 


11 


15 


10 


9 


15 


6 


13 


8 


2 


5 


145 


MN 


37 


29 


41 


37 


30 


27 


17 


24 


15 


17 


19 


13 


0 


306 


MT 


0 


0 


0 


0 


0 


0 


0 


15 


8 


13 


15 


3 


2 


56 


ND 


35 


43 


154 


158 


168 


128 


144 


163 


159 


89 


94 


73 


0 


1408 


NM 


197 


247 


363 


363 


369 


352 


353 


320 


347 


283 


256 


235 


126 


3811 


NV 


0 


0 


0 


0 


0 


0 


0 


3 


19 


11 


9 


10 


0 


52 


OR 


0 


0 


0 


0 


0 


0 


0 


0 


0 


30 


32 


43 


36 


141 


UT 


25 


9 


15 


22 


20 


23 


23 


0 


0 


0 


0 


0 


0 


137 


WA 


0 


0 


0 


13 


18 


11 


16 


10 


19 


22 


20 


12 


8 


149 


Wl 


0 


0 


51 


45 


54 


48 


51 


39 


45 


12 


16 


15 


1 


377 


WY 


18 


12 


16 


15 


10 


18 


11 


12 


5 


2 


5 


4 


0 


128 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 



READING 

State 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


AZ 


168 


246 


301 


302 


292 


262 


261 


239 


208 


146 


46 


30 


10 


2511 


IA 


0 


17 


21 


17 


12 


13 


22 


22 


14 


15 


17 


10 


0 


180 


KS 


9 


3 


4 


6 


8 


2 


3 


4 


1 


2 


6 


8 


8 


64 


ME 


0 


0 


12 


12 


17 


24 


22 


19 


19 


0 


0 


0 


0 


125 


Ml 


23 


12 


16 


10 


15 


10 


9 


15 


6 


13 


8 


2 


5 


144 


MN 


36 


28 


41 


36 


29 


27 


18 


23 


19 


20 


21 


14 


0 


312 


MT 


0 


0 


0 


0 


0 


0 


0 


16 


10 


15 


17 


3 


2 


63 


ND 


33 


43 


153 


157 


164 


126 


145 


163 


148 


89 


115 


89 


0 


1425 


NM 


199 


254 


364 


366 


366 


356 


353 


331 


342 


286 


262 


235 


121 


3835 


NV 


0 


0 


0 


0 


0 


0 


0 


3 


19 


11 


11 


11 


0 


55 


OR 


0 


0 


0 


0 


0 


0 


0 


0 


0 


27 


33 


43 


38 


141 


UT 


25 


9 


15 


21 


20 


23 


23 


0 


0 


0 


0 


0 


0 


136 


WA 


0 


0 


0 


4 


6 


2 


5 


20 


16 


36 


36 


19 


8 


152 


Wl 


0 


0 


56 


43 


54 


48 


51 


41 


45 


12 


16 


15 


1 


382 


WY 


18 


13 


17 


16 


9 


18 


11 


11 


6 


3 


6 


6 


0 


134 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 
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MATH 

ELO 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


Arizona Navajo Central ELO 


88 


92 


117 


134 


122 


118 


110 


67 


72 


82 


11 


9 


3 


1025 


Arizona Navajo North ELO 


0 


15 


75 


71 


58 


67 


67 


65 


66 


59 


34 


22 


10 


609 


Arizona Navajo South ELO 


0 


20 


14 


16 


14 


17 


11 


21 


16 


0 


0 


0 


0 


129 


Arizona South ELO 


11 


52 


47 


35 


40 


36 


36 


37 


26 


0 


0 


0 


0 


320 


Billings ELO 


18 


12 


16 


15 


10 


18 


11 


27 


13 


15 


20 


7 


2 


184 


Minneapolis ELO 


61 


58 


128 


110 


121 


105 


118 


114 


107 


57 


60 


40 


6 


1085 


New Mexico Navajo Central ELO 


104 


125 


152 


148 


138 


137 


127 


106 


107 


0 


0 


0 


0 


1144 


New Mexico Navajo North ELO 


116 


98 


108 


98 


115 


85 


84 


46 


32 


51 


47 


36 


38 


954 


New Mexico Navajo South ELO 


46 


49 


125 


133 


140 


142 


163 


165 


158 


151 


115 


99 


88 


1574 


New Mexico North ELO 


0 


0 


0 


0 


0 


0 


0 


49 


82 


81 


94 


100 


0 


406 


New Mexico South ELO 


0 


31 


43 


45 


50 


53 


41 


0 


0 


0 


0 


0 


0 


263 


Oklahoma ELO 


9 


3 


4 


6 


7 


2 


3 


3 


1 


2 


5 


7 


7 


59 


Sacramento ELO 


0 


0 


0 


0 


0 


0 


0 


3 


19 


11 


9 


10 


0 


52 


Seattle ELO 


0 


0 


0 


13 


18 


11 


16 


10 


19 


52 


52 


55 


44 


290 


Southern & Eastern States ELO 


0 


0 


12 


12 


17 


25 


22 


19 


19 


0 


0 


0 


0 


126 


Standing Rock ELO 


5 


22 


17 


14 


23 


6 


17 


10 


6 


0 


0 


0 


0 


120 


Turtle Mountain ELO 


30 


21 


137 


144 


135 


115 


108 


139 


126 


89 


94 


73 


0 


1211 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 



READING 

ELO 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


Arizona Navajo Central ELO 


99 


111 


118 


136 


123 


101 


110 


67 


69 


84 


11 


9 


3 


1041 


Arizona Navajo North ELO 


0 


15 


73 


74 


60 


67 


66 


67 


65 


62 


35 


21 


7 


612 


Arizona Navajo South ELO 


0 


22 


14 


17 


13 


17 


11 


22 


18 


0 


0 


0 


0 


134 


Arizona South ELO 


25 


51 


46 


36 


42 


35 


34 


37 


24 


0 


0 


0 


0 


330 


Billings ELO 


18 


13 


17 


16 


9 


18 


11 


27 


16 


18 


23 


9 


2 


197 


Minneapolis ELO 


59 


57 


134 


106 


120 


105 


119 


115 


111 


60 


62 


41 


6 


1095 


New Mexico Navajo Central ELO 


106 


135 


152 


149 


138 


139 


125 


107 


107 


0 


0 


0 


0 


1158 


New Mexico Navajo North ELO 


116 


99 


108 


97 


116 


86 


86 


46 


32 


51 


48 


36 


38 


959 


New Mexico Navajo South ELO 


46 


45 


126 


134 


136 


143 


164 


166 


152 


149 


114 


104 


83 


1562 


New Mexico North ELO 


0 


0 


0 


0 


0 


0 


0 


58 


83 


86 


100 


95 


0 


422 


New Mexico South ELO 


0 


31 


43 


46 


50 


53 


41 


0 


0 


0 


0 


0 


0 


264 


Oklahoma ELO 


9 


3 


4 


6 


8 


2 


3 


4 


1 


2 


6 


8 


8 


64 


Sacramento ELO 


0 


0 


0 


0 


0 


0 


0 


3 


19 


11 


11 


11 


0 


55 


Seattle ELO 


0 


0 


0 


4 


6 


2 


5 


20 


16 


63 


69 


62 


46 


293 


Southern & Eastern States ELO 


0 


0 


12 


12 


17 


24 


22 


19 


19 


0 


0 


0 


0 


125 


Standing Rock ELO 


3 


22 


18 


15 


24 


5 


17 


10 


6 


0 


0 


0 


0 


120 


Turtle Mountain ELO 


30 


21 


135 


142 


130 


114 


109 


139 


115 


89 


115 


89 


0 


1228 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 
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MATH 

ADD 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


ADD East 


44 


46 


170 


176 


182 


148 


150 


171 


152 


91 


99 


80 


7 


1516 


ADD Navajo 


354 


399 


591 


600 


587 


566 


562 


470 


451 


343 


207 


166 


139 


5435 


ADD West 


90 


153 


234 


218 


239 


223 


222 


240 


266 


216 


235 


212 


52 


2600 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 



READING 

ADD 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


ADD East 


42 


46 


169 


175 


179 


145 


151 


172 


141 


91 


121 


97 


8 


1537 


ADD Navajo 


367 


427 


591 


607 


586 


553 


562 


475 


443 


346 


208 


170 


131 


5466 


ADD West 


102 


152 


240 


208 


227 


213 


210 


260 


269 


238 


265 


218 


54 


2656 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 



MATH 

Project Status 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


Not NWEA Project School 


262 


269 


612 


651 


629 


604 


574 


582 


596 


589 


474 


395 


154 


6391 


NWEA Project School 


226 


329 


383 


343 


379 


333 


360 


299 


273 


61 


67 


63 


44 


3160 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 



READING 
Project Status 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


Not NWEA Project School 


268 


285 


616 


641 


616 


575 


566 


605 


586 


617 


526 


421 


149 


6471 


NWEA Project School 


243 


340 


384 


349 


376 


336 


357 


302 


267 


58 


68 


64 


44 


3188 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 



MATH 

School Type 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


BIE Boarding School 


139 


130 


224 


213 


244 


263 


256 


191 


192 


247 


140 


141 


119 


2499 


BIE Day School 


93 


153 


165 


183 


170 


123 


128 


97 


76 


0 


0 


0 


0 


1188 


Contract Day School 


0 


0 


12 


12 


11 


15 


9 


8 


13 


0 


0 


0 


0 


80 


Grant Boarding School 


110 


117 


214 


233 


224 


216 


225 


243 


285 


195 


175 


158 


48 


2443 


Grant Day School 


146 


198 


380 


353 


359 


320 


316 


342 


303 


208 


226 


159 


31 


3341 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 


Bureau 


232 


283 


389 


396 


414 


386 


384 


288 


268 


247 


140 


141 


119 


647 


Non-Bureau 


256 


315 


606 


598 


594 


551 


550 


593 


601 


403 


401 


317 


79 


1200 


Total 


488 


598 


995 


994 


1008 


937 


934 


881 


869 


650 


541 


458 


198 


9551 
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READING 
School Type 


K 


1 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Total 


BIE Boarding School 


144 


140 


224 


213 


247 


266 


259 


192 


193 


244 


140 


145 


118 


2525 


BIE Day School 


92 


158 


162 


184 


165 


122 


125 


97 


66 


0 


0 


0 


0 


1171 


Contract Day School 


0 


0 


12 


12 


11 


14 


9 


8 


13 


0 


0 


0 


0 


79 


Grant Boarding School 


119 


132 


212 


229 


214 


190 


212 


250 


274 


199 


183 


152 


45 


2411 


Grant Day School 


156 


195 


390 


352 


355 


319 


318 


360 


307 


232 


271 


188 


30 


3473 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 


Bureau 


236 


298 


386 


397 


412 


388 


384 


289 


259 


244 


140 


145 


118 


647 


NonBureau 


275 


327 


614 


593 


580 


523 


539 


618 


594 


431 


454 


340 


75 


1300 


Total 


511 


625 


1000 


990 


992 


911 


923 


907 


853 


675 


594 


485 


193 


9659 
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APPENDIX B: Paired Samples VCG Analysis 



MATH 


Spring RIT 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 


450 


0.589 


0.000 


149.2 


150.4 


-1.19 


-2.55 


449 


0.011 


1 


501 


0.750 


0.000 


168.7 


169.8 


-1.07 


-2.88 


500 


0.004 


2 


940 


0.669 


0.000 


183.2 


183.5 


-0.26 


-1.02 


939 


0.310 


3 


950 


0.759 


0.000 


193.3 


194.2 


-0.90 


-3.56 


949 


0.000 


4 


981 


0.728 


0.000 


201.2 


201.3 


-0.16 


-0.60 


980 


0.549 


5 


928 


0.787 


0.000 


207.4 


208.3 


-0.96 


-3.63 


927 


0.000 


6 


906 


0.796 


0.000 


213.0 


214.0 


-1.00 


-3.56 


905 


0.000 


7 


845 


0.821 


0.000 


217.0 


217.1 


-0.16 


-0.55 


844 


0.585 


8 


840 


0.840 


0.000 


222.7 


221.4 


1.34 


4.41 


839 


0.000 



MATH 


Raw Growth 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 


450 


0.313 


0.000 


15.3 


16.3 


-1.08 


-2.32 


449 


0.021 


1 


501 


0.426 


0.000 


14.6 


15.7 


-1.07 


-2.87 


500 


0.004 


2 


940 


0.427 


0.000 


11.6 


11.8 


-0.25 


-1.00 


939 


0.319 


3 


950 


0.398 


0.000 


10.5 


11.4 


-0.86 


-3.38 


949 


0.001 


4 


981 


0.329 


0.000 


8.9 


8.9 


-0.08 


-0.30 


980 


0.765 


5 


928 


0.141 


0.000 


7.1 


8.0 


-0.95 


-3.55 


927 


0.000 


6 


906 


0.191 


0.000 


6.2 


7.2 


-0.97 


-3.47 


905 


0.001 


7 


845 


0.081 


0.019 


4.9 


5.1 


-0.23 


-0.76 


844 


0.449 


8 


840 


0.089 


0.009 


5.7 


4.4 


1.36 


4.49 


839 


0.000 



MATH 


Norm Growth 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 




















1 




















2 


940 


0.279 


0.000 


-3.2 


-2.9 


-0.25 


-1.00 


939 


0.319 


3 


950 


0.314 


0.000 


-1.0 


-0.1 


-0.86 


-3.38 


949 


0.001 


4 


981 


0.177 


0.000 


-2.8 


-2.7 


-0.08 


-0.30 


980 


0.765 


5 


928 


0.075 


0.022 


-3.1 


-2.2 


-0.95 


-3.55 


927 


0.000 


6 


906 


0.145 


0.000 


-2.7 


-1.7 


-0.97 


-3.47 


905 


0.001 


7 


845 


0.073 


0.034 


-2.5 


-2.3 


-0.23 


-0.76 


844 


0.449 


8 


840 


0.039 


0.264 


-1.0 


-2.3 


1.36 


4.49 


839 


0.000 



40 | P a g e 



READING 


Spring RIT 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 


448 


0.525 


0.000 


148.6 


150.4 


-1.81 


-4.31 


447 


0.000 


1 


549 


0.763 


0.000 


164.1 


166.6 


-2.50 


-7.38 


548 


0.000 


2 


937 


0.752 


0.000 


178.6 


180.0 


-1.35 


-4.61 


936 


0.000 


3 


956 


0.773 


0.000 


186.2 


187.8 


-1.63 


-5.69 


955 


0.000 


4 


952 


0.728 


0.000 


193.8 


194.4 


-0.67 


-2.19 


951 


0.029 


5 


905 


0.743 


0.000 


198.9 


199.7 


-0.75 


-2.53 


904 


0.012 


6 


901 


0.747 


0.000 


203.1 


204.1 


-1.03 


-3.40 


900 


0.001 


7 


866 


0.715 


0.000 


207.3 


207.0 


0.28 


0.83 


865 


0.408 


8 


826 


0.778 


0.000 


210.5 


210.6 


-0.02 


-0.07 


825 


0.947 



READING 


Raw Growth 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 


448 


0.269 


0.000 


13.9 


15.7 


-1.78 


-4.24 


447 


0.000 


1 


549 


0.264 


0.000 


12.0 


14.4 


-2.43 


-7.21 


548 


0.000 


2 


937 


0.431 


0.000 


11.5 


12.8 


-1.34 


-4.55 


936 


0.000 


3 


956 


0.326 


0.000 


7.9 


9.5 


-1.60 


-5.59 


955 


0.000 


4 


952 


0.333 


0.000 


6.8 


7.4 


-0.65 


-2.12 


951 


0.035 


5 


905 


0.354 


0.000 


5.4 


6.1 


-0.72 


-2.41 


904 


0.016 


6 


901 


0.351 


0.000 


4.4 


5.4 


-1.00 


-3.31 


900 


0.001 


7 


866 


0.311 


0.000 


4.0 


3.7 


0.29 


0.87 


865 


0.383 


8 


826 


0.333 


0.000 


3.4 


3.4 


-0.02 


-0.07 


825 


0.948 



READING 


Norm Growth 




N 


Correla- 

tion 


Sig. 


Student 

Mean 


VCG 

Mean 


Paired 
Differen- 
ces Mean 


t 


df 


Sig. (2- 
tailed) 


K 




















1 




















2 


937 


0.275 


0.000 


-4.8 


-3.5 


-1.34 


-4.55 


936 


0.000 


3 


956 


0.150 


0.000 


-3.8 


-2.2 


-1.60 


-5.59 


955 


0.000 


4 


952 


0.079 


0.015 


-2.1 


-1.5 


-0.65 


-2.12 


951 


0.035 


5 


905 


0.086 


0.010 


-1.5 


-0.8 


-0.72 


-2.41 


904 


0.016 


6 


901 


0.103 


0.002 


-2.0 


-1.0 


-1.00 


-3.31 


900 


0.001 


7 


866 


0.010 


0.767 


-2.2 


-2.4 


0.29 


0.87 


865 


0.383 


8 


826 


0.033 


0.336 


-1.8 


-1.8 


-0.02 


-0.07 


825 


0.948 
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APPENDIX C: Virtual Comparison Group Methodology 



Virtual Comparison Groups 

NWEA researchers have developed a research control group methodology called Virtual Comparison Groups, or VCGs. 
Using VCGs, researchers can compare a student's academic growth (based on the difference between two MAP scores) 
to students who perform similarly and share similar demographic characteristics. Unlike comparisons using norms, VCG 
reports provide apples-to-apples comparisons. 

What is a Virtual Comparison Group? 

A virtual comparison group is a group of students that matches key characteristics of the group of students being 
compared. Researchers refer to it as a matched group quasi-experimental design. The comparison group is "virtual" 
because it is comprised of data from NWEA's longitudinal student achievement database that includes growth data from 
millions of students. In addition to student growth data, the database holds relevant district, school, and student 
characteristics, such as socio-economic status (SES) data, urban/rural designations, grade level, and beginning 
performance level. 

To construct a VCG, NWEA identifies the key characteristics of each student in the study group. The study group may be 
a class, school, or other logical organization of students. Next, for each student, NWEA filters the database to find 
students who perform similarly and share these key characteristics. A random sample of 51 comparison students is 
selected from that result and these become the Virtual Comparison Groups for each student in the initial study group. 
VCGs can provide valuable information for making program decisions or evaluating the effect of an educational 
intervention or policy. The VCG results control for variables that may otherwise mask the impact of the intervention. 
How are VCGs created? 

NWEA uses three types of filters to create a VCG for each student being studied. 

1. General Assessment Filters - Only students with valid test scores for the same year and subject area are 
considered. 

2. School Filters - The percentage of students qualified for free and reduced lunch is within 5 percentage points of 
the school being studied and schools have the same urban/rural classification (based on the National Center for 
Educational Statistics Common Core of Data Survey). 

3. Student Filters - Students are in the same grade level, within 1 RIT point of the beginning score, and the beginning 
and ending tests were taken within 7 days of the student's test date. This assures that the student and members of 
the VCG group had about the same amount of instructional time. 

After applying these three types of filters, a group of 51 students is randomly selected from the database. The median 
score of this group is the metric that is used to compare to the study group student score. 

If you are interested in learning more about how to use VCG data for your research study design, please 
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APPENDIX D: NWEA-BIE Partnership Backgrounder 



Backgrounder: 

Northwest Evaluation 
Association - Bureau of 
Indian Education 
Partnership 

NWEA is a not-for-profit organization located in 
Portland, Oregon with offerings in computer-adaptive 
assessment, research, professional development and 
reporting. Measures of Academic Progress (MAP ) 
assessments are aligned to state standards and can 
predict proficiency on state exams. 112 BIE schools in 
17 states use MAP assessments to inform instruction 
and predict proficiency on state tests. 

Through a professional development contract, NWEA 
works closely with School Leadership Teams and 
Education Line Officers in S3 schools and 17 line 
offices within the BIE's System of Support. Since the 
beginning of our work in September 2009. NWEA has 
delivered more than ISO workshops, 188 face-to-face 
leadership coaching events and 190 telephone 
coaching events to principals and Education Line 
Officers. "Roll-up" Reports with aggregated student 
achievement and growth data have been delivered to 
BIE at the conclusion of each testing season. 

The professional development and coaching 
plan is designed to work seamlessly with the BIE 
System of Support to provide structures and build 
capacity for the application of MAP student 
achievement data to improve effectiveness in BIE 
schools and make a difference for children The plan 
is designed to support and enhance BIE's 
"commitment to provide high-quality professional 
development opportunities" and help "ensure delivery 
of scientifically researched and evidence-based 
tnsTnimnnal nrartirps " 




NWEA 

Northwest Evaluation Association 

Partnering to http all kids learn 



The long term partnership between NWEA and the 
Bureau provides an environment for capacity building 
and proficiency in the use of student achievement 
data. The workshops delivered at schools are 
"leveled" over time to indicate a progression in 
complexity in the acquisition and development of skills 
and core knowledge. BIE personnel develop capacity 
to access and apply data to inform planning, 
implementation, and evaluation of student outcomes. 

To accomplish this work, NWEA works closely with four 
groups: 

1. Teachers 

2. School Leadership Teams, which include 
principals and may include teachers, 
instructional leaders, instructional 
coaches, and other leaders 

3. Education Line Officers 

4. BIE System of Support Management Team 

During the past year and a half, collaborative 
relationships between NWEA coaches and leaders 
have been established in most cases. The consistent 
support provided to BIE leaders is resulting in changes 
of practice in schools and line offices. Leaders are now 
eagerto set and monitor goals and have articulated, in 
many venues, the value of the coaching relationship. 
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In addition to changes in leadership practice, 
significant student growth in reading and 
mathematics achievement has been observed. The 
most outstanding instances include: 



Of the System of Support schools who tested tall 2009 
and spring 2010 (17 of 46 total): 

Outstanding Math yrnwth trn-n Fal T>9 tn Snnny '10 



Chilchinbeto grade 5: 11.8 RIT points 99% 

Chilchinbeto grade 6: 14.9 RIT points 99% 

Chilchinbeto grade 8: 9 RIT points 99% 

Nenahnezad grade 4: 9.S RIT points 99% 

Tiis Nazbas grade 4: 17.4 RIT points 99% 

Tits Nazbas grade 5: 14.1 RIT points 98% 

Chi-Chil-Tah grade 8: 13.8 RIT points 99% 



•'Nenahnezad and Tiis Nazbas both had more than 
typical growth in EVERY GRADE in Math 

Outstanding Read:ne growth from fall '09 to Soring 10 
Chilchinbeto grade 4: 17.8 RIT points 99% 

Chilchinbeto grade 6: 20.5 RIT points 99% 

Nenahnezad grade 4: 17.8 RIT points 99% 

Nenahnezad grade 6: 10.2 RIT points 99% 

Chl-Chil-Tah grade 8: 14.6 RIT points 99% 

Na Neelzhiin Ji Oita grd 4: 9.8 RIT points 
99% 

Ojo Endno grade 7: 13.9 RIT points 99% 

Alamo Navajo grade 6: 10.7 RIT points 99% 

Tohaali Community grd 5: 14.9 RIT points 
99% 



•‘Nenahnezad had more than typical growth In 
EVERY GRADE in Reading 



NWEA 

NrvVt-s.it Evaluation Auociolion 

IWnuruK «. Mf j# hj, luft 

121 Everett Street 
Portland, Oregon 97209 
Ta 503.624.19S1 
f AX 503.639.7873 
WWW.NWEA.ORG. 
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